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A DISTRIBUTED ARCHITECTURE AND ASSOCIATED PROTOCOLS 
FOR EFFICIENT QUAUTY OF SmiVICE-BASED ROUTE COMPUTATION 

Backbond of the Invention 

The present invention relates to the field of packet switcsfaing networks, 
e.g.. Asynchronous Transfer Mode (ATM) networks. Specifically, the invention 
relates to a distributed architecture and associated protocols for computing 
resource-sufficietxt routes capable of meeting a requested Quality of Service (QoS) 
level. The disclosed methodok)gies are equally applicable to oomxection-oriented 
netwoiks observing QoS-based source routing and connectionless networks 
supporting multiple QoS levels. 

R^-time multi-media appHcationd demand explicit QoS guarantees fiom 
the underlying network. To guarantee a given QoS level in the network, switching 
nodes or routecs ixoplement a QoS-based routing system that can observe expb'cit 
resource rescsrvation. Such a system endeavors to find oost-eSective rcssomco- 
sufficient routes. 

A QoS-based routing system requires netwoik nodes to gather and 
maintain netwoik topology as well as resource availability information. Figure 1 
shows such a network indicated generally by reference numeral 10, Network 
nodes 12» 14, and 16, wbi<*i arc routers or switches, exchange information, about 
their internal resourco availability including nodal-charactmstics, link-attributes, 
and end-system reachability infbnnation, etc. Databases 18, 20, and 22, 
connected to nodes 12, 14, and 16, respectively, store such network topology and 
resource availability information. A source node uses the network topology and 
resource availability information to compute resource-suffideat routes to the 
destination node that can support tlie desired QoS. 

As tho network gmws, there are correspoitkdit]^ increases in the size of the 
topology database, the time required to determine a route, and the network control 
traflBc oveAead. The network scalability problem associated with QoS routing 
has been addressed by the concq^t of hierarchical routing. A hierarchical routing 
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system divides the oetworic into logical gtou^. Only nodes that belong to (he 
53TOe logical group exchange detailed topology and link stale infbjmation with 
each other. A node outside the group only stores aggregate infoimatian about the 
group. 

To further scale down routing oveifaead xequirements as the mctwoijc 
grows, resource avallabilily information is exchanged only periodically (e,g., 
evefy several minutes) or at the occutr&Rce of a significant diange in resource 
availability (e.g., link/node Mlures, link utilization exceeding the 90% mark* ete.)- 
Because resource availability information may be outdated, an intermediate xH>de 
may reject a call setup request Xn this case^ the call Fstraccs its path to the 
originating node, which then tries an alternate route. 

Even in hierarchical networks, route computation requires considerable 
processing to find a cost-effective and resource^sufiScient path thai can meet the 
requested QoS level. First, the route computation engine at a node needs to 
perform a topology search to find a low cost path while ensuring that each X mk . 
(physicalor logical) in the path has the resources to accept tiiecozmectioEL In 
addition, topology database maintenance imposes considerable processing and 
storage requirements on switching nodes. Nevtttheless, currently implemented 
QoS-based and best effort routing networks requixe each node to compute routes, 
store and maint a in a topology database, and participate in the topology update 
process. The proposed architecture is tailored to distribute processing and 
memory requiremaits to support both QoS-based and best effort routing. 

A client-server architecture, in which a servw store$ all topology 
information and performs all route conq>utation, would alleviate the processing 
and storage requirements of each node. Figure 2 shows a centralized client-server 
network, indicated generally by re£»ence numeral 22, for implementing route 
computation. Under this approach, sitiglc route server 24 stores all network 
topology and resource availability information in associated database 26. Clients 
2&, 30, and 32 store no topology or routing information locally and therefore ixrust 
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communicate with server 24 every time they need to m^e rouling dedsioiis. 
Although the architecture in Figure 2 removes the processing and storage 
requirements at each node aa shown in Figure 1, network congestion still occurs as 
each client node must contact the server to make routing decisions. Also, limited 
processing power at the serv^ may impede fast computation of routes and 
communication of those joutes back to the client nodes. 

It is desirable, thetefbxc^ to provide a distributed architecture that makes 
intelligent use of processing elements distributed across the network to enhance 
performance of a routing system. Current QoS-based routing standards (e_g.. 
Private Network CO Networfc Xnterfece (PNNI) defined by the ATM Forum) do not 
address route computation methods. Earlier proposed route cadling methods 
(c-g^ routing in an Internet Protocol (IP) network), which store some route 
information locally, apply only to connectionless routing with no QoS guarantees. 
Cache tables in such schemes do not guarantee that the selected hop satisfies the 
requested QoS requirement It is even more desirable, therefore, to provide a 
client-server architecture for determining routes j&om a soiuce node to a 
destination address or node that can provide QoS guarantees along the route. 

The problem of route caching is more acute for networks that provide CJoS 
guarantees because^ in such an environment, the memory required to store cached 
routes may be prohibitively large. For example^ in a connection-oriented network 
observing QoS-bascd soxuoe routit^, a cached route entiy contains an entire 
description of an end-to-end path to a given destination for a given QoS level. On 
the other hand, a route that can satisfy a given QoS level may not be able to meet 
another C^oS requirement, and a source, therefore, may be required to store 
multiple QoS-bascd routes to the same destination. Similarly, a connocdonless 
network may be designed to store infbcmadon for the next hop, or route, for 
various QoS levels. Pltrthcnnore^ with d)iiainically changing resource availability 
inside the network, an end-to-rad route description or the next hop route entry for 
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a given QoS profile may not be valid after some time. Tberefore, the cached 
routes need to be i]pdated to ensure (heir validity. 

Earlier proposed dient-^sexver approaches to assist QoS^based routing (c.g.. 
Multiprotocol over ATM (MPOAX LAN Emulation (LANE), and Multicast 
Address Resolution Server (MARS) protocols) only perfomi address resolution 
and do not address distributed route computation methods- It is also desirable, 
therefore, to provide a distributed route computation methodology to complement 
these client-server architectures. Fuxtiier, there has been no effort to make use of 
distributed processing techniques to optimize perfomiance of a QoS-based routing 
eystem and enhance network scalability in such an envLconcoent. To further 
improve performance^ it is also desirable to eliminate the need for evety node in 
the network to participate in the topology exchange process and store network 
topology. 

Sxammv the Inventj^r^ 

This invcntiori satisfies those desires by providing a distributed client- 
server architecture that allows fast access to routing information* Route servers, 
distributed across the network, store and maintain a network topology database. 
Client nodes store pre-compuied routes locally and access route servers only when 
unable to obtain routing informatxoix locally. The present invention provides 
performance gains and mbances scalability in QoS4>ased networks observing 
source-based routing such as PNNI, MPOA, and LANE networks, 

A method consistent with the present invention comprises the steps of 
searching a route cache at a source node for a route satisQing a QoS profile and 
obtaining a route fixrni a route server if no route is found in the route cache. The 
method further oon^rises the step of updating the contents of the route cache 
based on network usage or changes m the state of the network. Anoth^cmethod 
consistent with the px^ent invention also con^)rise$ &e step of populating the 
route cache with a plurality of pre-comput&d routes. Yet anoth^ method 
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consistent with the ptescot invention comprises the stqps of seaxching a loute 
cache at a souzce node for a route satisfying a QoS profils, $oan^iing & route cache 
at a route server for a route if no route is found in the sourw rotite cache, and 
computing a route at the route scrvier if no route is found in the ^drv'er route cache. 

Apparatus and networks are also provided for carrying out methods 
consistent with the present invention. 

The advantages accruing to the present invention axe numerous. For 
example* rather than storing alt topology information on a ccsntralized server^ the 
invMtive architecture provides a subset of routing information at client nodes, so 
that a client node does not have to contact the route server on every call i^uest. 
A client node needs to contact a mute server only when information is not 
available locally. Further, each individual client node has the intelligence to leam, 
maintain, and adapt local information based on the statistical usage of the 
network, minimizing as much as possible the need to contact the route server. 
Advantageously, clients autonomously decide which subset of information to store 
locally. Finally, the invention provides protocols for synchronizing client 
information and the topology database stored at tiie route server. 

The above desires^ oth(^ desires, features, and advantages of the present 
invMition will be readily ^reciated by one of otdinary skill in the art fiom the 
following detailed description of the preferred implementations when taken in 
cormcction with the accompanying drawings. Both the foregoing general 
description and the following detailed description are exemplaty and explanatory 
only and do not restrict the claimed invention. The accompanying drawings, 
which are incorporated in and constitute a part of tbis specification, illustrate 
several embodiments of the invention and together with the description serve to 
escplain the principles of the invention. 
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Brief Description of the Drawings 
Figure 1 is a high level diagram of a fixlly distribute routing scheme 
known in the art; 

Figure 2 is a high level diagram of a prior art c«itralized architecture for 
route computation; 

Figure 3 illustrates a netwX)Tking scenario with distributed route caching 
consistent with the present invention; 

Figure 4 illustrates an implementation of distributed route computation 
consistent with the present invention; and 

Figures 5A-B illvtstratc the operation of a protocol for distributed route 
computation consistent with the ptcscnt invention. 

Detailed Description of the Prefe rrgd Bmbodlments 
Figure 3 illustrates a networic architecture consistent with the present 
invention in which networic topology databases are maintained and stored only at 
$elected route servere distributed across the networic For (his purpose, flic whole 
network is divided into clusters called routing tiers, three of which-^^ 40, 42, 
and 44— are shown for purposes of this discussion. Bach routing tier 40, 42, and 
44 includes a route server-46, 48, and 50, respectively-that services a group of 
clients distributed across an interconnecdon network. By way of example, route 
server 46 services clients 58, 60, and 62, route server 48 services clients 64, 66, 
and 68, and route server 50 sendees clients 70, 72, and 74. Wittiin the routing 
tiers, only the route s^ers store and maintain the cntne network topology, which, 
is stored, for example, in databases 52, 54, and 56. When the network updates the 
topology databases^ each server communicates with other servers to transfer 
information pertaining to its clients^ Client nodes do not maintain their own 
copies of the topology database. As will be described in more detail below, client 
nodes maintain local caches, not particulariy shown in Figure 3, to store pre- 
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cqmputed routes. Clients access servers to obtain route infoxmation only when 
pr&^mputod m^armatioii is not available at the clieafs local cache. 

Route severe A6f 48, mi 50 may be either network node$, such as xouteis 
ot switches, or s^arate processors which are not pait of the netwotk. The clients, 
on the other hand> may be ph3^cal switches or routers or processor elements 
within a distributed switching system. In a distnbuted route caching architecture 
consistent with the present invention, each dient communicates with its server via 
an intcrcormection network. The interconnection network can take nxany £:>rm$. 
With continuing reference to Figure 3. one sudi intetcoxmection network is shared 
bus 76, connecting clients 58, 60, and 62 to route server 46. Shared bus 76 may 
connect nodes within the same physical location or may operate on the backplane 
of a distributed switching syst^. A seccHid type of intercoxmection network is a 
X^cal Area Network (LAN) such as an Bthemet netwcitk;, an FDDI network, or a 
token ring network. For example, in Figure 3, 3LAN 80 intOTXjnnects client nodes 
70, 72, and 74 with route server 50. A third interconnection network. Wide Area 
Network (WAN) 78, interconnects dient iu>des 64, 66, and 68 with route server 
48. A distributed network caching methodology consistent with the present 
invention is not limited to a specific networking scenario or type of 
interconnection network, and the intercoxmection networks illustrated in Figure 3 
are exemplaiy only. 

When a network node receives a call setup request, it attempts to find the 
best route to the specified destination address or destination node. The selected 
route must meet the bandwidth requirements of the call as well as the required 
<JoS level. Routes are usually optimised based on u&er>speci£ied criteria, e.g., 
Administrative Weight (AWX end-to-end delay, and cnd-to-^d delay variation, 
either alone or in combination. 

A distributed network routing methodology consistent with the present 
invention selects optimal routes without duplicating (he entire topology database 
at each individual network node. Instead, each node contains a cache for storing a 
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set of prc-computed routes. These routes are classified with respect to their 
destioation addresses or nodes, baxKiwidth requirament, Viftual Private Network 
Identi£er, QoS level (e.g., eod-to-end delay lequirmieat, delay variance 
constraints, and cell loss ratio requirement), 3rul service category (eg., real-time 
or best effort, continuous bit rate or variable bit rate). These route attributes used 
for route classification axe collectively referred to herein as the QoS profile. 
When a source node receives a call destined for an address or riode for which a 
suitable route meeting the desired QoS profile is available in the cache, the node 
forwards the call along the route retrieved fiom the cache. If the client node 
cazmot find a suitable route in the cache, the client node solicits the route server to 
obtain a route to the desired destination. 

A route canhtng methodology consistent with the present invention 
reduces latency in route selection by providing from the cache a suitable route for 
an incoming call setup request. The number ofpre-computed routes that can be 
stored in the cache is limited by the amount of memory that can be dedicated to 
route caching at a single node. Furthermore, the pre-computed cached routes must 
be based on ^e latest view of the network topology stored at the route server. 
These constraints must be considered in determining the contents of each client 
node^s cache. 

Hxe following equation is uscfid in analyzing the dynamic of a distributed 
architecture consistrot with the present invention: 

Mean Route Access Time - A*h + (1) 
In equation il%A represents the average time required to find a suitable route in 
the client node^s local cache, B represents the average tune to fetdi a route fiom 
the route server when the client node cannot £nd a route locally, and /r represents 
the cache hit rate, defined as the £:action of requests for wiiich the client node can 
find a cached route. When the client node must contact the route server to obtain 
a route, the route server may have to perform route computation on demand, so 
that generally ^ is eignificantiy greater than ^. 
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Theiefoic^ one objective of a distzibut^ route cadiing scheme consistent 
vdth the present invention is to ma3dim2» h at each node. Tb& hit t^ale is 
maximized by cdcbiag a network conidor map at the client node, where the 
netwoik oomdor map is a table containing mutes fiom the client (source) node to 
every possible destination address for a given QoS profile. Caching networic 
corridor maps for multiple QoS profiles increases the hit rote further. A 
distzibated route caching scheme consistent vnlh the present invention allows 
network service providers to define QoS profiles for which routes should be 
cached based on their past experience with operating fhe network. 

While distributed route caching methodologies consistent with the present 
invention strive to maximize the bit rate, the memoxy constraints of storing pre- 
computed routes at the individual clients is also considered. Depending on the 
size of the network and the amount of cache m^noiy available at a client node^ it 
may not be feasible to cache a QoS-based route to every destination of local 
interest. When limited cache memory is available, climts cadie only routes to a 
subset of destinations of local interest with the QoS profiles of local interest For 
purposes of further discussion, a client node will be referred to herein as fiilly 
cached if the client caches QoS-based routes to all dostinations of local interest 
widi all the QoS profiles of local interest A. client node will be referred to hexein 
as partially cached if the cli^t caches routes only to a subset of network 
destinations of local interest Distributed route caching methodologies consistent 
with the present invention allow network service providers to define the 
destinations and QoS profiles of local interest- 
In a distributed route caching methodology consistent with the present 
invention, there are at least two alternatives for establishing the locality of interast 
at each client node: (1) static leamingi in which each client uses the locality of 
interest defined by the network service provider and does not modify the local 
interest based on network usage; and (2) dynamic learning^ in which each client 
uses the locality of interest defined by Uie network service provider to populate the 
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cache initially, but modifies the destinatioxts and QoS picfiiles of local intci:c3t 
based on netwoik iisage. With dynamic learning, the localiti<?5 of interest of the 
client nodes may vaty temporally (e.g,, by time of the day or day of week) as well 
as spatially (Le,, various nodes may commoidy request calls to di£E«:«nt sets of 
nodes or addresses in the network). Route caching consistent with the present 
invention exploits each node's pattern of past tCTipotal and spatial behavior to 
determine a locality of interest for the node at any point in time. By utilizing past 
bcihavxor» each node predicts which mutes it is likely to need in the near fiiture and 
stojces these routes in its cache. 

The peribrmance of distributed route caching schemes consistent with the 
present invention can be fiirther m3Xicni2:ed by reducing the latency, B, to fetch a 
route J&om the route serm- wh«i the client node cannot find a route locally. For 
this purpose^ in addition to maintaixung local caches at the client nodes, a 
distributed architecture consistent with the present inv^tion may maintain a cache 
at the server node as well. The cadbie at the server complements the cache 
information provided to the individual clients. For example, the individual clients 
may obs^c partial caching only for routes of local interest, while the server may 
use a full cache wbidi contains routes to all destinations with ail QoS profiles 
supported by the network service provider. 

A fall cache at the server consistent with the present invention caches 
netwoxk conidor maps with all QoS profiles siq^ported by the network service 
provider. On the other hand, a fully cached client stores <»ily routes to 
destinations of local intere^ for the QoS profiles of local interest Therefore, 
distributed toute cadbdng methodologies consistent with the present invention may 
operate in at least one of i^ur scenarios: (1) full nanh^T^g at both the server and 
cUcnts, and (2) full caching at the server and partial caching at the clients, (3) adl 
caching at the clients and partial caching at ^ server, and (4) partial caching at 
both the server and clients. 
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Thus, an an^tecture and me&odology consistent with the present 
invenfion determine the locality of interest of each client node, manage the limited 
cache space to suit the local interest, and synchronize the cache with the latest 
view of the topology database so that pre-compttted routes temain valid. 

By way of example only. Figure 4 details a specific software 
implementation of a distributed architecture consistent with the present invention. 
However, one skilled in the art will appreciate that there exist several 
iniplemeatations consistent with the present invention, and the invention is not 
limited to the specific impl^entation shown. Shared bus 100 mtercoimects route 
server 102 with client switching nodes 104, 10(>, and 108, all of which operate 
within one routing tier. Alternatively, the shared bus may be another type of 
interconnection network, e-g-* a LAN or WAN as discussed in connection with 
Figure 3. The client switching nodes may also be routers, and route server 102 
may be an additional switdiing node or may be a separate processor which does 
not route or switch traJQ&c. Each switching node 104^ 106, and lOS contains a 
route cache for storiri^ prc-computed routes 1 10, 1 12, and 1 14, respectively, a 
cache manager 116, 118, and 120, re^pecdvely, a caching agent 122, 124, and 126, 
teqE>ectivBly, and arouting agent 128, 130, arwJ 132, respectively. Route server 
102 includes network topology database 134, synchronizer 136, route dispatcher 
13S, background route computation engme 140, on-demand route computation 
engine 142, topology database manager 144, topology exchanger 146, and route 
cache 148. 

Cache managers 116, 118, and 120 may be software entities resident on 
each node. Eadi cache manager controls the usage of local memory available for 
route caching by using one or more of the following fimctions: (1) learning, Le., 
determining the population of die cache at start-up and which new routes should 
be added on an ongoing basis; (2) route replacement, i.e., determining which route 
entry should be removed from the cache when the cache is full and a new route 
entry xnust be added; and (3) route purging, Le., determining when and why 
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entries should be removed, separate fitwn the route replacement iunction. A client 
node can perfonn those operatiotis at a low priority, e.g^ the cliexxt xxiay 
accumulate multiple caching requests and bundle them togedier in a single 
message to the route server. 

Bach cache manago* determined the population of the c^^dbie using prior 
knowledge of the usage pattern of calls originating fiom the client node. The 
network service provider defuaes an initial QoS profile for each client node, 
characteri2»d by the destinaticm address, service cat^ory, bandwidth requirement, 
and required QoS level, or some subset thereof The cache manager may use the 
initial QoS profile to populate the cache with pxe-computed routes. 

The cache manager may inqreaso its knowledge of the node's locality of 
interest on an ongomg basis by tiackxi^ the call setup requests the node receives. 
When the client obtains a route fiom the server because its route cache contains no 
routes meeting the requested QoS pro&le and optimization criteria, the cache 
manager places the newly obtained route in its cache. Similarly, the cache 
manager updates the route cache as a result of crankback, a PNNI feature known 
in the art that enables a blocked call setup request to return to the source or an 
earlier intermediate node in its path, which attempts to reroute the calL When a 
node learns that a call setup has failed for a route previously selected by the node, 
the node mfbnns its cache manager to invalidate the route. If the node then 
obtains a new route fiom the server^ the cache manager rq>lacc& the new route in 
the cache in place of the old, invalid route. 

When the cache manager learns about a new QoS profile and attempts to 
add it to the client* s route cache, the cache manager may remove an entry to firee 
memory for the newly learned route. There are several c^roaches a cache 
manager consistent with the present invention may take, alone or in combination, 
in deciding whic^ entry to replace. According to one approach the cache 
manager maintains records of the last time an entry in 4ie cache has been hit, i.e.y 
the last time the client node selected the entry as an optimal route. With this 
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method^ the cache manager replaces the entry which has not been hit for the 
longest period of time. According to another ^proach, the cache manager 
maintains a coxiot of the numb^ of times an entry hai^ been hit, Le., the number of 
times the cache manager has selected the entry as an optimal route. With this 
method^ the cacbeiziaxiager replaces the oitry that has received the sma^ 
number of hits. According to yet a third £q)pn>acb, tlie cadie manager mflinfflitia 
both a count of the number of hits and the time elapsed since the last hit The 
cache maiiager then weights the count of hits by the time elapsed since each hit so 
that the w^ght of a particular hit on an entry decreases with time. In one 
implenientation of this weighted approach, an entry in the route cache has a 
holding priority Ht dxucing the Atb time interval represented by: 

^r*=w*i/,.; + (l-w)C4.,. (2) 

where t/*./ is the number of hits the given route received during the (A;-i)st 
interval^ and the weight IV is a number such that 0<=w<^l. Ifthecache 
manager must make a rqplacement decision during fhe time period the cache 
manager replaces the entry with the lowest value. If two or mote entries share 
the lowest value^ t}>c cache manager selects an entry to be replaced using some 
criteria* e-g., by randomly selcctixig one of the candidate entries. 

Even when the ca<^ manager docs not need to remove entries &om the 
cache because the cache is underutilized, it may be desirable to purge any cached 
routes that have remamed in the cache for a significant period of time. Similarly, 
when a route suffers a crazild>ack» it may be purged. 

Consistent with the present invention* the client node may puirj^e routes 
autonomously or with assistance from the route server, as will be described below. 
In one approach to autonomous purging^ the client node places an upper limit on 
the lifetime of a cached route, r^ardless of how often the node obtains the route 
directly fiom tlie route cache. Upon expiration of the route lifetuno, the cache 
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maxiager purges the route from the cache, A pur:ged route may be aatomatically 
fe^ed again from the route server if certain criteria are met, e-g., if the puigcd 
route was popular in the recent past, as measured by companng its hit rate to a 
predefined hit rate threshold. 

With continuing reference to Figure 4, caching agents 122» 124. and 126 
may be software entities resident on each client node. Each caching agent stores, 
rieanoves, and ietneve$ cached iputes on a demand basis. Each caching agent is 
optimised for efHci^t retrieval of the cached routes based on the destination 
address, QoS profile, and optimization criteria. For eTcample, each caching agent 
may use a data structure th3t can efficiently seanrfi the cache to find aioute that 
satisfies the QoS requirements and optimization criteria to a given destination. In 
addition to matching QoS profiles with routes in the lo<:;al route ca<^c, the caching 
agent may impl«nent some additional functions. For example, if for a given 
destination address and QoS profile there are multiple loutes available in the 
cache^ the caching agent may achieve load balancing by using a technique such as 
weighted round robin to select one route over others based on certain criteria. 
Another function of the cadiing agent is to detennine whether to use a cached 
route whose QoS profile surpasses that of the call request in order to avoid 
cozitactixig the route senr^ to obtain a route. For example, the caching agent may 
decide to use a cached route capable of si^porting 2 Mbps bandwidth even though 
the connection requires only 1,5 Mbps bandwidth. 

Routing agents 128, 130, and 132 are software modules on each node that 
interpret a call request to extiact the destination address or node and QoS profile. 
Based on this information, arouting agent pr^Kues a route request message for 
the caching agent. If a route caimot be obtained from the cache, the routing agent 
solicits the route segrver for a suitable route. When the routing agent obtains a 
route from the route sexvear, the routing agent informs the cache manager so that 
the cache manager can learn the new route and add it to the client nodc^s route 
cache. 
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Synchronizer 136 in route searrar 102 maintains syxichionization of 
networtc topology database 134 and route caches 1 10, 112, and 1 14 at the client 
nodes. For this purpose, the synchronizer tracks usage of routes by the distributed 
clients. If the client nodes impl^ent autonomous puigixig, then, consistent with 
the present invention, route server 102 tracks route usage by remembering, for a 
period of time equal to the lifetime of the ixtutc befom purging occurs, that a 
particular client node has stored a route in its route cache. If the route undergoes a 
change before the route lifetime expires (c.g.. if a node becomes inactive and must 
be removed fiom all routes, or if cached routes become invalid as a result of a 
resource availability update or crankback and no longer satisfy Hhe required QoS 
profile), the route server informs the client of the change, instmcts the client to 
purge the route, and discontinues updates for that route to the client. If the client 
determines that it should continue to store the route in its route cache, it fetches 
die route again fiom the route server. Alternatively, instead of instructing the 
client to purge the route, the route server can provide the client with the newly 
computed route, which would remain in the clients route cache for tho lifetime of 
the route. In either case, once the route lifetime has CTcpired, the route server 
assumes that the client has aheady purged the route, and discontmues updates 
pertaining to that route. If the cli^ has already purged or replaced the route 
before the route lifetime has expired, the client will ignore the route server's 
updates. Hiis overall approach, in which the route server rmi^bers which routes 
are stored in each client's route cache, ensures that locally stored routes are always 
srynchronized with the route server's view of the network, as stored in its network 
topology database. 

In a second method for tracking route usage Consistent with the present 
invention, the client docs not perform autonomous purging of routes fiom its route 
cache. Instead, the client depends on Che route server to track die routes stot^ in 
the local route cache and to refiesh them periodically. The route server maintains 
its knowledge of routes in the local route caches either by periodically probing the 
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clients or by being infomicd by the clieats when an existing route is replaced. 
Under the fii^ sccaario, there may be a p^od of time between probes during 
which the local caches may not be synchronized with the route s«vcr. Under the 
second scenario, however, the local caches are synchronized with the route 
server's topology database. The second approach, however, requixos more 
processing by both the client node and the route $erver. 

When a client node cannot find a route in its route cache that satisfies a 
given call setup request, the routing agent of the client contacts the jtoute s^er to 
obtain a route. Route s^^ 102 recdves all routmg requests through route 
dispatcher 138, which is responsible for instructing route computation engines 140 
and 142 to compute routes in the bacl(@rouod or on d^nand, re$^tively. 
Background route conoputation caigine 140 typically confutes network corridor 
maps or routes currently cached by the client nodes to populate individual clients 
on aperiodic basis. On-demand route computation engine 142 typically computes 
a route when a client cannot iEind a route locally and contacts the route server to 
obtain a route. 

When route dispatcher 138 receives a request for a route, it forwards this 
request to on-deniand route computation agent 142. Alteinativeiyi if route server 
102 also maintains its own route cache of pre-computed routes, as illustrated by 
route cache 148 in Figure 4, route dispatchwr 138 may search the server's route 
cache before soliciting arx on-demand route computation. Maintaining 
indepwdent cache 148 at the server is advantageous when the client nodes have 
limited memoiy and/or the scrvw has a large memoty space for route cadiing. 
With its own route cache, the server may store network conidor m^ containing 
optimal routes to all destinatioDS with sevoal QoS profiles to avoid on-demand 
path computation as rnuch as possible. Server route cache 148 is maintained in 
tiie same manner as other candies in the system. Wheth^ route dispatcher 138 
obtains a route fiom a computation en^e or tbe server's route cache, it returns to 
the client a cost-«ficcttve and resource-sufficient route to the given destination. 
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Topology database manager 144> shown in Figure 4, maintains network 
topology database 134 stored at route server 102 and receives updates from 
topology exdianger 146. Route servei$ ajoross a netwoik exdiange infomiation 
throu^ topology exchanger entities, which typically participate in the topology 
update process on behalf of the clients in its routing tier. This an;hitecture reduces 
the processing needed at client nodes by removing them fiom the topology update 
process. 

Figures 5A-B illustrate the operation of a distributed route coinputation 
protocol consistent with the present invention tmdcr two scenarios. In Figure 5 A, 
the desired route is obtained firom the local cache, and in Figure 5B, a desired 
route is not available in the local cache. In Figure 5 A, the client node receives a 
nsute request at its routing agent, Tbe muting agent then pr^ares a route request 
for the cadiing agent, including the destination address, QoS profile, and 
optimization criteria. The caching agent retrieves a suitable route fiom the local 
route cache and informs the cache manager of the route hit so that the cache 
manager can maintain statistics of route usage. The caching agent delivers the 
retrieved route to the rputiixg agenl^ which passes the route to the switch or router. 

Figure SB illustrates retrieval of a route firom the route server. The routing 
agent receives a route request and passes the destination address, QoS profile, and 
optimization criteria to the caching agent If the cadiixiig agent cannot £nd a 
suitable route in the local cadie, the caching agent mfonns the routing agent of the 
failure. The routing agent then contacts the route server through the 
interconnection networic (e,g., shared bus, LAN, or WAN), passing the desstination 
address^ QoS profile, and optimization criteria to the server. The route server 
obtains a suitable route, which it transnaits back to the routing agent in the client 
The routing agent infom[i& the cache manager and caching ^ent of the new route 
60 Qiat the newly obtained route can be added to the cache- Fmally, the routing 
agent delivers the route to the switch or router. 
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It will be appreciated by t}K>se skilled in this ait &at various modifications 
and variations can be made to the distributed architecture and protocols consistent 
with the present invention described herein without departing irom the spirit and 
scope of the invention. Other embodiments of the invention will be £q[)parent to 
those skilled in this art fiom considesration of the specification and practice of the 
invention disclosed herein. For example, Figuxe 4 illustrates only an exemplary 
implementation of software modules, and many other implementations consistent 
with the present invention are possible. In the methodology discussed in 
connection with Figure 4« clients observe partial caching and dynamically learn 
about the locality of interest based on network usage. However, several 
combinations of caching (full car partial at the clients and/or s^ver) and cache 
learning (static and dynamic) are consistent with the present invention. For 
example, when destination addresses and QoS profiles are static, the cache 
manager does not use learning algorithms. Shnilarly, if a cliont node stcr^ routes 
to every possible destination of local interest for die QoS profiles of interest, tlie 
cache manager need no replace routes to fiee memory for storing newly learned 
routes. Instead, the client simply replaces the old entry with the newly learned 
cxitry. Additionally, handling of crankbacks may be simplified where the client 
node labds the route which has suff^ed the iailure and either uses cm alternate 
route ftom its own cache or requests the server for an altflamative route. If anew 
route is obtained fiom the server, the client node simply updates the labeled route 
»wth the newly learned entry. The server, on the other hand, may trigger 
tecomputation of the network corridor mc^, possibly in the background, in the 
event of a crankback failure and/or on a periodic basis. The route server may 
bundle updates for a v^le network oorddor map xn one message to simplify the 
&3mchroni2ation process, Therefoie, it is intended that the specification and 
examples be considered exemplary only, with the true scope and spirit of the 
invention beua^ indicated by the following claims. 
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We claim: 

1. A method, for use in a packet &witz;hins network, for selecting a 
route between a source node and a destination node, whexein the route satisfies a 
QoS profile^ wherein the source node stores in a route cache a plurality of pre- 
computed routes originating at the source node« and whearein the souroo nodo is 
connected to a route server, the method comprising steps of: 

searching the route cache for a route satisfying the QoS profile; and 
if no 8&&sfymg route is found in the toute cache, obtaining a route fiom the 
server satisfying the QoS profile, 

2. The method of claim 1 further coraprising the stqp of: 
updating the contents of the route cache based on networic usage. 

3. The method of claim 2 wherein the step of updating includes: 
adding the route computed at the server to the route cache. 

4. The method of claim 3 wherein the step of updating further 
includes: 

if the route cache is full before the comptiied route is added^ removing one 
of the pre-computed routes fbom the route cache. 
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5, The method of claim 3 \^cnsm the step of updating further 
includes: 

for each px«-computed route in the route cache* measuring time since the 
pre-computcd route was selected as the route between the source node and the 
dcstmation node; 

oomparmg the measured times; 

if the xx>ute cache is full before the computed route is added, choosing one 
of tibie prB-coiii|>uted routes to be removed fifom the route cach^ based on the 
comparison of measured times; and 

rc^ovin£; the chosen one of fhc pre^computed routes horn the route cache. 

6. Tbe method of claim 3 wherein the step of updating further 
includes: 

for each pre^computed route in the route cache, maintaining a count of the 
number of times apr^-com^uted route is selected as the route between the source 
node and the destination node; 

comparing the counts; 

if the route cache is full before the computed route is added, choosing one 
of the pre^4x>mputed routes to be remove from the route cache based on the 
comparison ofcounts; and 

removing the diosen one of the pre-computed routes from tlie route cache. 
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7. Hie method of claim 3 whcrdn the step of updating furfhcr 
includes: 

for each pre^mputed Toutc in the route cache, measuiing time since the 
pre-computed route was selected as the route between the soutce node and the 
destination node; 

for each pre-computed route in the route cache, maintaining a count of the 
number of times a pre-computed route is selceted as the route between the souroe 
node and llie destination node; 

weighting the counts by the measured times; 

compaiing the weighted counts; 

if the route cache is full before the computed route is added» choosing one 
of the pre-computed routes to be removed from the route cache based on the 
comparison of weighted counts; and 

removing the chosen one of the pre-computed routes fcom the route cache. 

8. The method ofclaim 7 wdierein the step of weighting the counts by 
the meastired times includes the substep of 

calculating a holding priority diuing the kth time interval according to &e 
equation H|( = w*Hk.i + (1-w) XSi^u where U^,, is the count during the (k-l)st 
interval, and w i s a number such that 0 w <>= 1 ; 

and wherein the st^ of choosing one of the pre-compxitcsd routes to be removed 
includes the substep of 

during (he time period choosing the one of the pre^-computcd routes with 
the lowest Hp. 

9. The method ofclaim 1 farther comprising the step oft 
purging a pre-computed route fitom the route cache. 
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10. The method of diaim 3 wherein the step of updating further 
includes: 

for each prc-computed route in the route cache, znea^oriug dme smc& Cbe 
pre-computed route was added to the route cache; 

choosing one of the pre^<xnnputed routes to be purged fiom the route cache 
based on the measured time^ aud 

purging the chosen one of the prc-con^uted routes &om the route cache. 

11. The method of claim 3 wherein the step of updating finther 
includes; 

for each pre-<x)mputed route in the route cache, measuring time since the 

pre-computed route was added to the route cache; 

comparing the measured times to a pTedetennined route lifetune; and 
puiiging from tilie route cache the pre-compnted routes whose measured 

time exceeds the route lifetime. 

12. The method of claim 1 1 wherein the step of updating fiitthcr 
includes: 

for each pre-computed route in the route cadie, maintaining a count of the 
number of time& a pre-computed route is selected as the route between tiie somce 
node and the destination node; 

i^>datxng the purged route based on a current state of the netwoiic; and 
adding the updated route to the route cache if the count before purging 
exceeds a threshold. 

13. The method of claim 1 vdierdn the route server maintains a 
database of the network topology, the method further comprising the step of: 

synchroxuzing the route cache with the network topology database. 
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14. Hxe method of claim 13 herein the step of synchronizing includes 
the substep of: 

the route server probing the source node to determine the contents of the 
route cajche. 

15. The mediod of claim 1 furttier comprising the step of: 

updating the routes stoied in the route cache in response to a change in the 
netwotk. 

16... The method of claim 1 further comprising the step of: 
updating the routes stored in the route cache in response to a &ilure in the 
network. 

17. The method of claim 1 further comprising the step of: 
updating the routes stored in the route cache in response to a crankback in 

the nctwork- 

18, Tlie method of claim 1 further comprising the step of: 
t^Kiating the routes stored in the route cache in response to a change in 

availability of a resource in the network. 
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19, A method, for use in a packet switching netwoxk, for sclocting a 
route between a source node and a destination node, wherein the route $ati$£ies a 
QoS profile, and wherein the source node is connected to a route server^ the 
method comprising the st&ps of: 

populating a loule cache at the source node with a plmiality of pre- 
computed xt>utes originating at the source node, 

searching the route cache for a route satisfying the QoS profile; and 

if no satisigring route is found in the route cache, obtaining a route from the 
server satisi^ing the QoS profile. 
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20. In a packet switchiog netwoik mcluding a plurality of mute ^ccvcfsi 
and a plurality of cUmt netwotk nodes, wherein each client node is connected to a 
route server, wherein each client node stoics in a route cache a plurality of pre- 
computcd routes originating at the client node> wherein the route server stores in a 
5 server route cache a second plurality of pre^mputcd routes, a method for 

selecting a route between a Si^t node and a aecond node that satisfies a QoS 
profile, the method comprising Qxc steps of: 

searching the client route cache for a route satisfying the QoS profile; 

if no satisfying route is found in the client route cache, searching the server 
10 route cache for a route satisfying the QoS profile; and 

if no satisfying route is found in the server route cache, computing a route 
at the server that satisfies the QoS profile. 
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21 . In a packet switching netwoiic including a plurality of n>ute eervers 
and a plurality of client netwoxk nodes, wherein each client node is connected to a 
route server, and wherein the route server maintains a database containing network 
topology and rcsouxcc availability infonnation to bo used for computing routes 
that satisfy a QoS level, a method for updating the database coix^jrising the stops 
of: 

eadi route server maintaining network topology and resource availability 
information; and 

each route serv^ exchanging with the remainder of the plurality of route 
servers network topology and resource availability information about the clients 
councctod to it. 
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22. A packet switching network coroprismg: 
a route server; 

A source node coimoctod to the route serv^ 
a dostiuation node; 

5 ^ xoute cache connected to the source node, the route cadie containing a 

plurality of pre-computcd routes originatins at the source mode; and 

means for selecting a route between the source node and the destination 
node that satisfies a QoS profile, the means comprising: 

means for searching the route cache for a route satisfying to the 
10 QoS profile; and 

means for obtaining a route from the server that satisfies the QoS 
pro^e if no $atisfying route is found in the route cache. 

23. The network of claim 22 further comprising means for i^xJatiag the 
route cache based on network iisage. 

24. The network of claim 23 wherein the means for updating includes: 
means for adding the route computed at the server to iho route cache. 

25. The network of claim 24 wherein the means for updating further 
includes: 

means for removing one of the pre-computed routes from the route cache if 
the route cache is full before the computed route is added. 
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26, The n(3twoik of claim 24 whwein the means for upcbting further 
inoludes; 

m(?ao$ Sar measuring Umo since each pre-computed route wa3 sclectod as 
the route between the source node and the destination node; 
means for comparing the measured times; 

means for choosing one of the pre-compntcd routes to be removed fiom 
the route cache based on the comparison of measured times if the route cache is 
full before the computed route is added; and 

mean$ for removing the chosen one of the pre-computcd routes from the 
route cache, 

27. The network of claim 24 wherein the means for updating fiirthcr 
includes: 

means for maintaining a count of the number of times each pre^mqputcd 
route is selected as the route between the source node and the destination node; 
means for comparing the counts; 

means for choosing one of tlie pre-computed routes to be removed &om 
the route cache based on the comparison of counts if the route cache is fiill befoje 
the computed route is added; and 

means for removing the diosen one of the pre-computed routes &om the 
route cache. 



CA 02256937 1998-12-23 



- 29 - 

28. The netwDifc of claim 24 wherein the means for updating further 
includes: 

means for measuring time since each pre-computed route was selected as 
the route bctweott the soxuce node and the destination node; 

means for maintaining a count of the number of times each pro-conq>uted 
route is selected as the mute between the source node and the destination nod€^ 

means for weighting the counts by the measured times; 

means for con^aring the weighted counts; 

means for choosing one of the prc-compntcd routes to be removed ftom 
the route cache ba^ed on the comparison of weighted counts if the route cache is 
foil before the computed route is added; and 

means for removing the chos^ one of the pre-computed routes from the 
route cache. 

29. The actworic of claim 28 wherein the means for weighting the 
counts by the measured times includes 

means for calculatmg a holding priority during the fcth time interval 
according to the equation H = w*Hfc.i -I- (1-w) where Uk-, is the count during 
the 0£:-I)st interval, and w is a nuznber such that 0 <= w 1; 
and wherein the means for choosing one of ^ pre-computed routes to be 
removed includes 

during the time period p, means for choosing the one of the pre-con^tcd 
routes with the lowest Hp. 

30. The network of claim 23 forther comprising: 

means for purging a pre^mputed route fix>m the route cache. 



CX 02256937 im- 12-23 



- 30 - 

3L The network of claiw 24 wfierdn the means for updating further 
includes: 

for each pro-computed route in the route cache, means for measuring time 
since the pre-computed route was added to the route cache; 

means for choosing one of the pre-computed routes to be pmgcd Bx>m the 
route cache based on the measured time; and 

means for purging the chosen one of the pre-computed routes from the 
route cache. 

32. The network of claim 24 wherdox the means for i^dating further 
includes: 

for each pre-computed route in the route cache» means for moasuriii^g time 
since the pre-computcd route was added to the route cache; 

means for comparing the measured times to a predetarmined route 
lifetime; and 

means for piffgunig fiom the route cache the pre-compuied routes whose 
measured time exceeds the route lifetime. 

33, The network of claim 32 whcr^ the moans for updating further 
includes: 

for each pre-computed route in the route cache* means for maintaining a 
count of the number of times a pte-computed route is selected as the route 
between the source node and the destination node; 

means for updating the purged route based on a current state of the 
network; and 

means for adding the updated route to the route cache if the count before 
purging exceeds a threshold. 
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34. The nctworic of claim 23 wherein the route sars^er maintains a 
database of the networic topology, the netwoifc further comprising: 

means for synchronizing the route cache with the network topology 
database. 

35. The network of claim 34 wherein the means for synchronizing 
includes; 

means for probing the source node to deteraune the contents of the mute 

cache. 

36. The netwoik of claim 22 further comprising: 

means for updating the routes stored in the route cache in response to a 
change in the network. 

37.. The network of claim 22 further comprising: 
means for updating the routes stored in the route cache in response to a 
failure in the network. 

38. The network of claim 22 further comprising: 

means for updating the routes stored in the route cache in response to a 
crankback in the network. 

39. Thenetworkof claim 22 further comprising; 

moans for updating the routes stored in the route cache in respotiso to a 
change in availability of a resource in the network. 
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40. A packet switdbditig network coiuprising; 
a route server, 

a source node connected to the route servw; 
a destixiation node; 
5 a route cache conxu^xsted to the source node; 

means for populating the route cache with a plurality of pre-coxnpuCed 
routes originating at the source node; and 

means for selecting a route betweotn the source node and the destination 
node that satisfies a QoS profile, the means comprising: 
10 mxsan^'for searching the route cache for a route satisfying to the 

QoS profile; and 

means for obtaining a route fiom the server that satisfies the QoS 
profile if no satisfying route is j^und in the route cache. 
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4h A packet switdbing network comprising: 
a plurality of route servers; 

a plurality of client network nodes, each client node connected to a route 

serven 

a route cache connected to eadi client nodc» the route cache containing a 
plurality of pre-comput{!)d routes originating at the chent node; 

a route cache connected to each route server containing a second plurality 
of pre-coinputed routes; 

means for selecting a route between a first node and a second node that 
satisfies ft QoS profile^ the means comprising: 

means £or sean^ng the client route cache for a route satis^ing the 

QoS profile; 

means for searching the server route cache for a route satisJ^ng the 
QoS profile if no satisfying route is found in the client route cache; and 

means for computing a route at the server that satisfies the QoS 
profile if no satisfying route is found in the server route cache. 
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42. A packet switcbing network comprising; 
a phirality of route stivers; 

a pIxuBlity of client network nodes, eadti client node connected to a route 
server; and 

a network topology and resource availability databa^ used to compute 
routes that satisfy a QoS level, connected to cacsh route server, 
wherein each route server comprises; 

means for inaintaimng network topology and resource availability 
information; and 

means for exchangizig topology and resourDe availability information 
pertaining to the clients connected to the route server with the remainder of the 
plurality of route servers. 
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43 . A packet switching netwoifc comprising: 
a toutc server; 

a pluxality of source nodes connected to the route server; 

a pliirality of destination nodes; 
5 a source route cache connected to each source node, the source route cache 

containing a pre-comput^ route fix>m the source node to each destination node of 
local interest; and 

means for selecting a route between the source node and the destination 
node that satisfies a QoS profile, the means comprising: 
10 means for searching the source route cache for a route satisfying 

the QoS profile; and 

means for obtaining a route fiom the server satis^ing the QoS 
profile if no satisfying route is found in the route cache. 
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44. A packet switching netwoik comprising: 
a route server; 

a plurality of souxro nodes connected to ibp route server; 
a plurality of destination nodes; 

a source route cache connected to each source node, the source route cadie 
contaiwng a pnx^omputed route Grom the source node to a destination node in the 
networfc; 

a server route cache connected to the route server, the server route cache 
containixig a route &om each source node connected to the xoute server to each 
destination node in the network; and 

means for seleotixiig a route between the soiuce node and ^e destination 
node that satisfies a QoS profile, the means con^iising: 

means for searching the source route cache for a route satisfying 
the QoS profile; and 

means for obtaining a mute fix)m the server satisfying the QoS 
profile if no satisfying route is found in the route cache. 
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45. A packet switching netwoik comprising: 
a route server; 

a plurality of source nodes connected to the mute server 
a plurality of destination nodes; 

a source route cache connected to each source nodc« the source route cache 
contaix^ung a pie-computed route fincm the source xiodc to a destination node in the 
network satisfying each of aplurality of QoS profiles of local intensst; and 

means for selecting a route between the source node and the destination 
node that satisfies one of the plurality of QoS profiles, the means comprising: 

means for searching the source route cache for a route satis^ing 
(he QoS profile; and 

means for obtaining a route from the server satisfying the QoS 
profile if no satisfying route is found in the route cache. 
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46. , A packet switching network comprising; 
a route server; 

a plurality of source hcmJcs coitnected to the route server; 
a plurality of destitiatioa nodes; 

a soutce route cadie connected to each souice node, the source route cache 
contaixung a pre-computed route ftona the ^urcc node to a destination node; 

a server route cache connected to iJie route a«rver, tfie server route cache 
cQjitaimng a route from a source node connected to the route server to a 
dfistitxation node in the netwoifc satisfying each of a plurality of QoS profiles 
provisioned in the network; and 

means for selecting a route hetween the source node and the destinatioa 
node that satisfies a QoS profile^ the means comprising: 

means for searching the source route c^dttti for a route satisfying 
the QoS profile; and 

mcaiig for obtaining a route finom the server satisfying the QoS 
profile if no satisfying route is found in the route cache. 
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In a packet switching xxetwotk, a distributed axtshttecture provides efficient 
computation of routes in Quality of Service (QoS>-based rDUting scenarios. Using 
a client-server model, only designated route servers store and maintain a database 
containing the entire network topology, so that each network node is not rxsquired 
to store and maintain the network topology. Client nodes maintain a cache 
containing pre-computed routes so that they can oflen make touting decisions 
autonomously. A client contacts a designated route server only when the client 
cannot obtain tow. its local cache a route to a given destination that meets the 
performance requiremeiits. A client cache may contain pre-computed routes with 
designate QoS profiles to all destinations or to a subset of destinations. Route 
servers may also contain caches, which may contain prc^^^n^uted routes to all 
destinations in the network with aU QoS profiles, or may contain only a subset of 
such routes. 

Bach client node may also be provided with intelligence to leam» maintain 
and adapt local information based on the statistical usage of the notwoik. CUent 
caches may learn statically, i,e, the ozchc contains routes based on a QoS proOle 
provided by the network service provider, or they may leam dynamically, Le., 
routes are modified based on ongoing network usage statistics. The goal is to 
minimize the need to contact the route server as much as possible. Protocols are 
defined to maintain synchronization between the route server and its clients 
distributed across the nctwoik. These protocols need to be observed to ensure that 
all nodes have the latest view of the network topology stored at the route server. 
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